IK>COHEIT WSOBB 



BD 20B Oft 

&OTHOB 
TITLE , 

FOB D&TI ' 
HOTE 



EDKS PRICE 
DESCRIPTORS 



IDENTIFIERS 



TM 810 7tl1 

Halpln, Gerald; And Others « / 
Research on irltlng: k Searph for 01).jecti?e Beasures 
Related to Hollstlcally Scored Essays. 
Bar 81 . - ^ 

13p.; Paper presented at the Annual Beetxng of the 
Eastern Edacatlonal Research Association 
(Philadelphia r PA^ Barch, 1981) • , ^ . 

BF01/PC01 Plus Postage. 

Correlation; ^Essay Tests; Higher Education; Holistic 
Evaluation; "^Holtiple choice Tests; l^stxng Problems; 
♦Test falidity; writing (Coapositian) ; iriting 
Research; ♦Writing Skills 
Bissouri College English Test 



ABSTRACT ' ^ . 

Controversy has characterized the research on 
writing. On the one side are those who state that, to evaluate 
achievement in writing, evaluate the wtiting of students. Of{ the 
other side are those pointing out the problems associated vxth direct 
measures of writing such as low reliability and higii cost in terms of 
time and often dollars. The purpose of this study was to determine i^ 
selec^\objectively measured indirect . indicators of writing ability do 
account for scores obtained on hoiistically scored direct measures ol 
writing ability. The indirect measure was the Bissouri College 
English Test, composed of 90 multiple* choice items assessing 
punctuation, capitalization, grammar, spelling, sentence stjfle and 
structure, and paragraph organization. The direct measure was an 
essay on a general topic, to be written in 30 m^u^eis. Bivariate and 
multiple correlational analyses were conducted between each sujbtest 
of the Missouri t^st and the essay.- All six independent measures 
together accounted for 26 perctot of the variance in the essay 
scores. These results^ight make questionable the sole use g£ 

taking decisions about competence in written ^ 



indir^t measures 
English, (l^thor/3 
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' RESEARCH ON WRITING: A SEARCH FOR OBJECTIVE 
MEASURES RaATED TO'HOLISTICALLY SCORED ^SAYS 

'tears ago, Diederich (1946) said' that "the essay is unquestionabi^ a 
valid test of ability to write, foi^ it is an in§,tance, a sample, of the very 
ability that one is attempting to mei^re. There is.no mo.re direct evidence 
of ab'ility to write" fp. 584). Eley (1955) '.added that "an adequate essay 
test of writing is valid by definition; that is to say,. it has face validity 
si-nce it requires' the candidate to perform the actual behavior which is being 
measured" (p. -11'). Brown (cited in Palmer, 1961). summed up^arguments such as ' • 
these and others favoring the essay by saing: "Everybody agrees that writing 
should be tested by writing" (p. 472). However, l^jbtai nifig rel iable j^ates , 
of ^wHtHig ability via the essay continues to -be. an expens'ive, time-consuming 
- process (Coffman, 1971, Lutz, Note 1 ^ Note 2). -That writing samples must be 
obtained on multipl'e occasions and scored by several raters to obtain very 

reliable estimates of students' true ^ilities has been def/nstrated in a num- 

/ 

ber of studies (Braddock, Lloyd-Jones, & Schoer, 1963 ,'lCoffman; 1966, Godshalk. 
Swineford, & Coffman, 1966, Llabre, 1978; McColly, 197(1). 

Indirect measures of writing ability have arisen ^s an alternative to 
direct asse'ssments requiring that essays be written ar>d scored. These indi- 
rect ' assessments require no writing at all but instead require that examinee's 
respond to. related questions cast in a multiple-choice fonnat. Indirect meas- 
ures usually have greater reliability and consequently tend to be favored by 
those concerned with qual it/measurement. Arguing for indirect assesment, ' 
Noyes., Sale, and Stalnak^r (1945) suggested that 'a student assessed by means 
of an essay is somewhat in the position of a gambler ^ho risks all on a single 
throw of the dice, while a multiple-choice test, allows many throws: "The ^ood 
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1^ candidate who errs on a few of these (multiple-choice items) has plenty of 
opportunity to redeem himself; a mistake on one item does not affect apy other 
item. In writing a theme, however, th^ candidate who makes a false start al- 
most inevitably involves his whole theme in difficulties even though he may be, 
generally speaking, a good writer" (p. 9). Objecting to the assessment of 
writing skill via- multiple-choice tests, though', Braddock, Lloyd-Jones, and- 
:Schoer (1963) wrote that "not only da they (multiple-choice tests) not require 
the examinee to perform the actual behayior be>^9 measured-he does no actual ■ 
writing, but these tes^s also make little or no attempt to measure the M^j>^r 
elements' of composition, even indirectly" (p. 42). ^ 

It i 5. apparent from the literature that controversy hascX^acterized the 
research on writing. On the one side are those agreein^j^i^th McCaig (1977) 
who stated that to evaluate achievement in writi j^val uate the writing of 
•students. On the other side are those pointi^ut the problems associated 
with direct measures'of writing such asJJ^ reliability and high cost in terms 
of time and often dollars. Due to diff icul ties with direct assessment of 
writing, -many practitioners hav^nsequently opted to use'indirect measures.,, 
Akeju (1972), for example,ylncluded thai the reliability of the West African 
General Certificate Eduj^ion examinations in English composition v^*^ inade- 
quate and suggested Jffe use of multiple-choice tests. The College Board has 
used objective ( i^. . multiple-choice) English tests for years (Palmer, 1961). 
Before object^ tests of English competence are used in lieu of writing 
samples,, hyfever, it is important that they be demonstratively related to 
essays, ^hey should indeed, considering the charges of Braddock, Lloyd-Jones, 
* and Sil(!er (1963), measure the various elements of composition. 
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The purpose of this study was to determine if select objectively measured . 
indirect indicators- of writing ability do 'account f^r scores obtained on 
holistically s-CQred Kju-ect measures of writing ability. Specifically investi- 
gated were the J)i vari ate and multivariate relationships between the independent 
or predictor variables of punctuation, capitaTizati orr, grammar, spelling, sen- 
tence style and structure, and. paragraph organization as measured by the Missouri 
College English Test and the dependent or criterion variable of essay score 
obtained via the holistic scoring procedure. 

' _ ' Procedure 

Participants in this study were 172 sophomores and juniors who^were ^ppli- 
cants for admission to teacher education at a large state university who had to 
pass an English prbficiency examination prior to acceptance into tttie teacher 
education program. Since participation in this- English proficiency testing 
program is required, all applicants were examined thereby .preventi ng a selection 
problem due to non-cooperation. Both direct and indirect assessments ^ere made. 

# 

The indirect measure of writing ability was the Missouri College English 
Jest (Call is & Johnson, 1965), an objective measur^ composed of 90 multiple- 
choice items assessing competence in six-areas: punctuation, capitalization, 
grarmiar, spelling, sentence style and structure, and paragra'ph organization. 
It was administered to the subjects in a monitored, test-like condition in a 
large university auditorium at a time designated for the testing. Student re- 
sponses were electronically scored for a. total score and scores on the six 
subtests of punctuation, capit,al ization, grammar, spelling, sentence style and 
.structure, and paragraph organization. The resulting raw scores were usisd in ^ 
the analyses. ^ 
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• ,,Internar consistency alpha reliability coefficients were computed for the 
tbtal and for each subtest. ,As shown in Table 1, the reliability estimates 



Insert Table 1 About Here' • 



ranged 'from .55 to .73 for the subtests and was .80 for the totj^l . These re- 
liabilities were generally low but sufficient for researdh purposes. 

The direct measure of writing ability was an essay which the subjects wrote 
after completing- the Missouri English test. The tl^c they were given to write 
about was very'general (e.g., friendship), and they were instructed to narrow 
the topic d.own and write the essay in 30 minutes. 

Three faculty members who had professional preparation for and experience 
l-^n'^eaching English jind/or English education at the University leve,T were chosen 
to evaluate the writing samples. The holistic method of scoring was utiljzed. 
This process is based upon a generalized impression or global quality of a paper 
Based upon the reconnendation of Coffman (1971) a 10-point scale was chosen. 

The three judges were"brought together and given'a 1-hour training session 
on scoring procedure. Nfxt, they rated seven sampl'e essays which were not part 
of the 172 essays being used as data in this study. An average interrater 
reliability .82 Wfcs obtained for the seven sample papers using the analysis 
of variance method recommended by Ebel (1979). The average ratings assigned 
by the three raters were also very close: 6.14, 6.14, and 7.05r7^*Pftqtively. 
With this kind of agreement, the raters proceeded to rate the l)2-essays whjch 
were the data source for this study^. After all 172 papers were scored by all 
three raters, the average interrater reliability was again computed. The 
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resulting coefficient was .86 (see Table 1). The average ratings given, by the 
three raters were also most congruent: 5.'81, 6.20, and 6.66, v^pecti vel-y. 



Data Analyses and Results ' 

Bivariate. and multiple correlational analyses were conducted between each 

sub^st of the Missouri test and ^the essay (score = average' of the three judges' 

ratings) using stepwise regression procedures of the Statistical Parckage for 
r 

the Social Sciences (Nie, Hull, Jenkins, St$inbrenner, & Bent,^ 1975). A Pearson 
r was also computed between^ the Missouri test total and the average essay rat- 
ings, (See Table 1 fo'r raw score means and standard deviations.) 

• .J 

Results of the correlational analyses are shown in Table 2. As can be seen 
in this table, the tivariate between e^ch subtest on the Missouri test and * 
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the essay ranged from .25 to ,40 (£ < .01)/ The multiple ^ using all six 
Missouri subtests was ,51 ^£ < ,01) which was very close to the bivariate cor- 
relation between the total Missouri (sum of subtest scores) and the essay, r = 
.49, £ < .01. 

When the bivariate correlations were corrected for unreliability of' the 
indirect measTyres, the resulting theoretical correlations ranged fron^.31 to 
.54 (see Table 2). Although a reliability estimate of the essay was not avail- 
able '(the reported interrater reliability is actually a measure of objectivity 
of scoring^ the observed rel iabil ity'for the essay in this study would be much 
less than unity (1) if essay reliabilities from other studies are generalized. 

With such being the case, the true relationship between the indirect and direct 

f 



measures in this study would have further increased if a cc^rrection for atten- 
tuation (unreliability) in the essay measure could* have also'been made. 

Discussion 

This study suggests, as have previous studies (cf. Breland & Gaynor, 1979; 

* 

Hogan & Mishler,* 1980i Crocker, Ondrasik, & Lamme, Note 3), that' indirect and ' 
direct measures of writing ability-tend to tap simjlar skills. All six of ^the- 
objective measures considered accounted for a significant portion of the vari- 
ance in writing performance in this study. - Frbm a bivariate perspective these 
predictors accounted for 5% to 16% of the v'ariabi 1-ity in writing performance. 
All six independent measures together accounted for 26% of the variance which^ 
is only 2% more variance -accounted for than was 'obtained by simply using a 

V 

total Missouri score. 

From the opposite perspective, however, the^ix indirect objective meas- 
ures in combination leave about 74% of the variability in writing unexplained. 
Each of the indirect measures expl ain^cKsjome of t'he individual "differences in 
the direct measure of writing, but the correlations, -although significant, were 
small. From an educational perspective, the fact that tbese indirect measures 
failed to account for a larger percentage of the variability in the writing 
sample might make questionable the sol$ use of such measures for making deci- 
sions about competence in written English. Better educational decisions might 
be made using a combination of both direct and indirect measures of writing 
proficiency , jpatikularly since the raters in this stxjdy seemed to be able to 
tap a broad spectrum of writing abilities in their holistic scoring instead of 
allowwig specific variables such as grammar and spelling that are meas^ufrable 
objectively to dominate tfieir ratings. i 
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Means, Standard Devn'atT^ns, and Reliability Coefficients for th'e 
Mi5^ouri College English Test and the Essay Test 



Test 


X 




SD 


Rel iability 


Missouri College English Test 










Punctuation 


7. 


58 


2.10 


.55*" • ■ • 


Capital ization 


8. 


03 


2.52 


.67** 


Grammar 


10. 


70 


2.76 


.59* 


Spelling 


9. 


76 


2.58 


^ .59* 


' Sentence^ Style and Structure 


6. 


33 


2.05 


.56* • 


Paragraph Organization 


14. 


03 


5.80 


.73* . ' 


TOTAL 


59 


.11 


11.80' 


r .80* . v' 


Essay . 


18 


.61 




.86** 



Note. " * X aljDha internal consistency reliability 
** = average interrater reliability 
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\ Table 2 

f MuUip1e-Cor»^at1ons,|R' Squape', Increase in R Square, 
.Pearson Correlations. Correctad Correlations, and Standardized Beta Wei g^ts 
• for Direct 'andr Indirect Measures of Writing. 



Variable 



Multiple R 
R Square 



Increase -in 
R Square 



Pearson Corrected . Standardized 
r r Beta 



— ^ 

Punctuation 




.40* 


*^16 • 


' .16 


.40* 


.54 


.20* ^ 


Spelling ^ 


\ 


.46* 


.21, . 


* .05 


.39* 


.52 


'.17*' 


Paragra'pfh 
Organization 


r 


.49* 


.24 


.03 


.26* 


.to 


.14* ^ 


Sentence Style 
and Structure 




.50* 


.25 


.01 


.35^ 




.11 • 


• 

Grammar 




.51* 


>• 

.26- 


.01 


.35*- 


.46 


.10 . 


Capitalization 




.51* 


.26 


. .00 ^ 


.25* 


'.31 


.04 


Missouri Total 






• 




.49* 


' .55 




£ < .01. 




A— 

'■X. 
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